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3-D SELECTION AND MANIPULATION 
WITH A MULTIPLE DIMENSION HAPTIC INTERFACE 

Field of the Invention 

[0001] This invention relates generally to manipulating virtual objects in a modeling 

environment. More particularly, the invention relates to the selection and manipulation of three- 
dimensional objects represented in a two-dimensional display space. 

5 Background of the Invention 

[0002] Systems and methods for selecting and manipulating objects in computer-based 

modeling environments are known. Examples of existing systems that select and manipulate 
objects include circuit board design systems and systems used for designing mechanical objects, 
such as automobiles and airplanes. Some systems typically impose restrictions on a user, such as 

1 0 placing objects such as circuit elements on predefined planes. Other systems require a user to 
specify a location of a component with high precision in three dimensions and three angular 
orientations. The devices commonly used for providing information to the modeling system 
include alphanumeric input devices, such as keyboards, and cursor-based devices operating in 
two dimensions, such as computer mice and trackballs. Present computer-based modeling 

1 5 systems generally do not permit a user to readily modify an object that has been selected. 

[0003] The stringent rules that present computer-based modeling systems impose and the 

inability of users to select, manipulate, and modify objects in a manner more akin to natural 
motion limits the ability of users to create models of three-dimensional objects. 

Summary of the Invention 

20 [0004] The systems and methods of the invention provide a user the ability to select 

three-dimensional virtual objects in a three-dimensional modeling environment using two- 
dimensional representations of the objects. In broad overview, the invention involves a 
multidimensional degree of freedom haptic interface that controls a three-dimensional cursor. A 
user employs the cursor to select an arbitrary point on a three-dimensional virtual object on 

25 interest. The user can select the object by positioning the cursor so as to coincide with a point of 
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the object in two of three dimensions, and issuing a select command. Through the application of 
a mathematical transformation, the system determines the difference between the position of a 
selected point on the object and the position of the cursor. The system displays the cursor at the 
location of the selected point on the object, so that the user can more easily manipulate or edit the 
5 object in the modeling environment. When the user releases the object after manipulation is 
completed, the cursor is relocated to the position it would have had the manipulations been 
applied to the cursor directly. 

[0005] In one embodiment, the invention features a method for selecting an object in a 

three-dimensional modeling environment. The method includes the steps of generating a three- 

10 dimensional modeling environment containing one or more virtual objects and a three- 
dimensional cursor, determining a first three-dimensional cursor position in the three 
dimensional modeling environment, the three-dimensional cursor position corresponding to a 
position of an input device having at least three degrees of freedom, representing a first view of 
at least one of the one or more virtual objects in a first two-dimensional display space, 

1 5 representing the three-dimensional cursor position in the two-dimensional display space, and 
selecting one of the virtual objects based on a positional correspondence of the object and the 
cursor in the two-dimensional display space. 

[0006] In some embodiments, the input device has at least six degrees of freedom. In one 

embodiment, the input device has exactly six degrees of freedom. In some embodiments, the 

20 virtual object is a selected one of a point, a straight line segment, a curvilinear segment, a spline, 
a two-dimensional representation, and a three dimensional representation. 
[0007] In some embodiments, the method further includes the step of editing the selected 

virtual object. In some embodiments, editing the selected virtual object is a selected one of 
coloring the object, modifying the object, combining the object with another virtual object, 

25 grouping the object with another object, deleting the object, and renaming the object. In some 
embodiments, editing the selected virtual object includes sculpting the selected virtual object. 
[0008] In some embodiments, the method further includes the step of performing a file 

function involving the selected virtual object. In some embodiments, performing a file function 
includes saving the object to a file. 
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[0009] In some embodiments, representing a first view of at least one of the one or more 

virtual objects includes representing the one or more virtual objects in a selected one of a 
perspective view and an orthogonal view. 

[0010] In some embodiments, the method further includes the steps of selecting a local 

5 origin point on the selected virtual object and defining a mathematical transformation in the 
three-dimensional modeling environment, the mathematical transformation representative of the 
difference in location of the local origin point and the three-dimensional cursor position. 
[0011] In some embodiments, the local origin point is an arbitrary point on the object. In 

some embodiments, defining the mathematical transformation includes defining a vector having 

10 a component directed orthogonal to the two-dimensional display space. In some other 
embodiments, defining the mathematical transformation includes defining a mathematical 
transformation having at least one of a three-dimensional translational vector, a rotation about the 
local origin point, and a rotation about the three-dimensional cursor position. 
[0012] In some embodiments, the method further includes the steps of applying the 

1 5 transformation and manipulating the virtual object, the manipulation of the virtual object 
corresponding to a manipulation of the input device by the user. 

[0013] In some embodiments, the method further includes the step of manipulating the 

virtual object, the manipulation of the virtual object corresponding to a manipulation of the input 
device by the user combined with an application of the transformation. In some embodiments, 

20 the manipulation of the input device includes at least one of a translational degree of freedom and 
a rotational degree of freedom. In some embodiments, the manipulation of the input device 
includes a simultaneous manipulation of two or more independent degrees of freedom. In some 
embodiments, the manipulation of the input device includes a simultaneous manipulation of three 
or more independent degrees of freedom. In some embodiments, the manipulation of the input 

25 device includes a simultaneous manipulation of six or more independent degrees of freedom. 

[0014] In some embodiments, the method further includes the step of relocating the three- 

dimensional cursor to the location of the local origin point by application of the mathematical 
transformation. In some embodiments, the relocating step is performed only during the duration 
of the manipulation. 
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[0015] In some embodiments, the method further includes the step of providing a visual 

aid to help the user select and manipulate the virtual object. In some embodiments, providing the 
visual aid includes providing a user-activated constraint limiting a point to a locus aligned to an 
axis of the three-dimensional modeling environment. 
5 [0016] In some embodiments, the method further includes the step of moving the three 

dimensional cursor to a position it would have if the manipulation of the input device by the user 
had been applied directly to the three dimensional cursor. In some embodiments, the moving 
step is performed upon a command issued by the user. In some embodiments, the command is a 
release of the selected virtual object. 

10 [0017] In some embodiments, the method further includes the step of providing a visual 

aid to help the user select and manipulate the virtual object. In some embodiments, providing the 
visual aid includes providing a user-activated constraint limiting a point to a locus aligned to an 
axis of the three-dimensional modeling environment. In some embodiments, providing the visual 
aid includes providing a context-specific visual aid consistent with user-defined geometrical 

15 limitations. In some embodiments, providing the visual aid includes representing a second view 
of at least one of the one or more virtual objects in a second two-dimensional display space, the 
first two-dimensional display space and the second two-dimensional display space corresponding 
to different planes of the three-dimensional modeling environment. In some embodiments, 
representing the second view includes representing the second view on the second two- 

20 dimensional display space whose plane is orthogonal to a plane of the first two-dimensional 
display space. 

[0018] In some embodiments, the input device is a haptic device. In some embodiments, 

the haptic device includes a haptic device providing force feedback to actuators operating in at 
least three degrees of freedom. 

25 [0019] In some embodiments, the method further includes the step of providing a haptic 

aid to help the user select and manipulate the virtual object. In some embodiments, the haptic aid 
includes the provision of dynamic friction force during the positional correspondence of the 
object and the cursor in the two-dimensional display space. In some embodiments, during an 
activation of the user-activated visual constraint limiting a point to a locus aligned to an axis of 

30 the three-dimensional modeling environment, the haptic aid includes a haptic constraint limiting 
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motion of the three-dimensional cursor to directions aligned to an axis of the three dimensional 
environment, except within a region of radius R about an identified point. 
[0020] In some embodiments, the method further includes the step of contemporaneously 

displaying a visual aid component that indicates an axial location of the cursor along an axis. In 
5 some embodiments, during an activation of the user-activated visual constraint limiting a point to 
a selected line, the haptic aid includes a haptic constraint limiting motion of the three- 
dimensional cursor to the line. 

[0021] In some embodiments, the method further includes the step of contemporaneously 

displaying a visual aid component that indicates the location of an axial location of the cursor 
10 along an axis. In some embodiments, during an activation of the user-activated visual constraint 
limiting a point to a selected plane, the haptic aid includes a haptic constraint limiting motion of 
the three-dimensional cursor to the plane. 

[0022] In some embodiments, the method further includes the step of contemporaneously 

displaying a visual aid component that indicates the location of the plane. 

1 5 [0023] In one aspect, the invention relates to an apparatus that permits a user to select an 

object in a three-dimensional modeling environment. The apparatus includes a computer that 
supports a three-dimensional modeling environment application, an input device that provides 
user input to the computer, the input device having at least three degrees of freedom, a modeling 
module that, when operating, generates the three-dimensional modeling environment using the 

20 computer, the three-dimensional modeling environment adapted to model one or more virtual 
objects and to employ a three-dimensional cursor, and a selection module responsive to user 
commands that, when operating, selects one of the virtual objects based on a two-dimensional 
positional correspondence of the object and the cursor. 

[0024] In some embodiments, the apparatus further includes a display device that 

25 provides a two-dimensional display space for presenting to the user representations of the virtual 
object and the three-dimensional cursor in the modeling environment. 
[0025] In some embodiments, the apparatus further includes a rendering module that, 

when operating, renders on the display device a view of the virtual object in a selected one of a 
perspective view and an orthogonal view. In some embodiments, the input device has at least six 
30 degrees of freedom. In one embodiment, the input device has exactly six degrees of freedom. In 
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some embodiments, the virtual object is a selected one of a point, a straight line segment, a 
curvilinear segment, a spline, a two-dimensional representation, and a three dimensional 
representation, 

[0026] In some embodiments, the apparatus further includes an editing module that, 

5 when operating, edits the selected virtual object in response to user input. In some embodiments, 
the editing module is a selected one of a module that sculpts the object, a module that colors the 
object, a module that modifies the object, a module that combines the object with another virtual 
object, a module that groups the object with another object, a module that deletes the object, a 
module that renames the object, and a module that performs a file function involving the object. 

1 0 [0027] In some embodiments, the apparatus further includes a cursor tracking module 

that, when operating, determines a position of the three-dimensional cursor in the three 
dimensional modeling environment, the position of the cursor corresponding to a position of the 
input device, an object tracking module that, when operating, tracks a local origin point on the 
selected virtual object, and a transformation module that, when operating, defines a mathematical 

1 5 transformation in the three-dimensional modeling environment, the mathematical transformation 
representative of a difference in location of the local origin point and the three-dimensional 
cursor position at a time the user selects the virtual object. In some embodiments, the 
transformation module defines the mathematical transformation in terms of at least one of a 
three-dimensional translational vector, a rotation about the local origin point, and a rotation about 

20 the three-dimensional cursor position. 

[0028] In some embodiments, the apparatus further includes an object manipulation 

module that, when operating, manipulates the virtual object, the manipulation of the virtual 
object corresponding to a manipulation of the input device by the user combined with an 
application of the transformation. In some embodiments, the object manipulation module 

25 represents the manipulation of the input device using at least one of a translational degree of 
freedom and a rotational degree of freedom. In some embodiments, the object manipulation 
module is adapted to manipulate at least two independent degrees of freedom simultaneously. In 
some embodiments, the object manipulation module is adapted to manipulate at least three 
independent degrees of freedom simultaneously. In some embodiments, the object manipulation 

30 module is adapted to manipulate at least six independent degrees of freedom simultaneously. 
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[0029] In some embodiments, the apparatus further includes a relocation module that, 

when operating, relocates the three-dimensional cursor to the location of the local origin point by 
application of the mathematical transformation. In some embodiments, the relocation module is 
operative only during the duration of the manipulation. 
5 [0030] In some embodiments, the apparatus further includes a visual aid module that, 

when operating, provides a visual aid to help the user select and manipulate the virtual object. In 
some embodiments, the visual aid module is responsive to a user command, the visual aid 
module constraining a display of a point manipulated by a user to a locus aligned to an axis of the 
three-dimensional modeling environment. 

1 0 [0031] In some embodiments, the apparatus further includes a cursor repositioning 

module that, when operating, moves the three dimensional cursor to a position it would have if 
the manipulation of the input device by the user had been applied directly to the three 
dimensional cursor. In some embodiments, the cursor repositioning module operates in response 
to a command issued by the user. In some embodiments, the command is a release of the selected 

15 virtual object. 

[0032] In some embodiments, the apparatus further includes a visual aid module that, 

when operating, provides a visual aid to help the user select and manipulate the virtual object. In 
some embodiments, the visual aid module is responsive to a user command, the visual aid 
module constraining a display of a point manipulated by a user to a locus aligned to an axis of the 

20 three-dimensional modeling environment. In some embodiments, the visual aid module is 
responsive to a user command, the visual aid module constraining a display of a point 
manipulated by a user to a locus consistent with user-defined geometrical limitations. In some 
embodiments, the visual aid module represents a second view of at least one of the one or more 
virtual objects in a second two-dimensional display space, the first two-dimensional display 

25 space and the second two-dimensional display space corresponding to different planes of the 
three-dimensional modeling environment. In some embodiments, the visual aid module 
represents the second view on the second two-dimensional display space whose plane is 
orthogonal to a plane of the first two-dimensional display space. 
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[00331 I n some embodiments, the input device is a haptic device. In some embodiments, 

the input device is a haptic device having force feedback actuators operating in at least three 
degrees of freedom to apply force to the user. 

[0034] In some embodiments, the apparatus further includes a haptic aid module to help 

5 the user select and manipulate the virtual object. In some embodiments, the haptic aid module 
computes a dynamic friction force to be applied to the user by way of the haptic device during a 
positional correspondence of the object and the cursor in two dimensions of the three- 
dimensional modeling environment. In some embodiments, during an activation of the user- 
activated visual constraint limiting a point to a locus aligned to an axis of the three-dimensional 

10 modeling environment, the haptic aid module activates the force feedback actuators to provide 
haptic force to the user upon deviation of the point from the locus. In some embodiments, the 
visual aid module is additionally adapted to display a visual aid component that indicates a 
location of the cursor along an axis. In some embodiments, during an activation of the user- 
activated visual constraint limiting a point to a selected line, the haptic aid module activates the 

1 5 force feedback actuators to provide haptic force to the user upon deviation of the point from the 
line. In some embodiments, the visual aid module is additionally adapted to display a visual aid 
component that indicates a location of the cursor along a line. In some embodiments, during an 
activation of the user-activated visual constraint limiting a point to a selected plane, the haptic 
aid module activates the force feedback actuators to provide haptic force to the user upon 

20 deviation of the point from the plane. In some embodiments, the visual aid module is 

additionally adapted to display a visual aid component that indicates a location of the cursor on a 
plane. 

[0035] The foregoing and other objects, aspects, features, and advantages of the invention 

will become more apparent from the following description and from the claims. 
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Brief Description of the Drawings 
[0036] The objects and features of the invention may be better understood with reference 

to the drawings, described below, and the claims. The drawings are not necessarily to scale, 
emphasis instead generally being placed upon illustrating the principles of the invention. In the 
5 drawings, like numerals are used to indicate like parts throughout the various views. 

[0037] Fig. 1 is a schematic perspective view of a force reflecting haptic interface; 

[0038] Fig. 2 illustrates a general purpose computer that can be used in accordance with 

one embodiment of the present invention; 

[0039] Fig. 3 is an illustrative flow diagram that shows the steps of one embodiment of 

10 the method of the invention; 

[0040] Fig. 4 shows an exemplary three dimensional virtual object that is generated in a 

three-dimensional modeling environment, along with a three dimensional cursor, according to 
principles of the invention; 

[0041] Fig. 5 shows an illustrative example of a circumstance wherein the user brings the 

1 5 position of the interface into collocation with an intended point of an object; 

[0042] Fig. 6 shows an illustrative example in which the user has controlled the position 

of the interface so as to bring the three-dimensional cursor near coincidence in two dimensions 

with a point on a virtual object, according to principles of the invention; 

[0043] Fig. 7 shows an illustrative example of the operation of the methods and systems 

20 of the invention when the three-dimensional cursor reaches the location of a point of a virtual 

object; 

[0044] Fig. 8 is a diagram that shows an illustrative manipulation of a virtual object 

combined with application of a transform, according to principles of the invention; 
[0045] Fig. 9 is a drawing that illustrates an embodiment in which a selection of a virtual 

25 object is depicted when a display is operating in an orthogonal view mode, without perspective, 
according to principles of the invention; and 

[0046] Fig. 10 is a drawing that illustrates an embodiment in which a selection of a 

virtual object is depicted when a display is operating in a perspective view mode, according to 
principles of the invention. 
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Detailed Description 

[0047] The methods and systems of the invention involve the use of a two-dimensional 

representation of objects that exist in a three dimensional modeling environment. One 



embodiment of a two-dimensional representation is the use of a computer display that presents 
5 objects using only two dimensions, such as rows and columns of pixels. A three-dimensional 
modeling environment is a computer application program that accepts as input, or constructs, one 
or more virtual objects which undergo computational operations in a computational space 
analogous to the real physical space that one is accustomed to. The computational operations 
represent physical operations that are being considered for performance on an object in real 



environment before carrying out similar operations in reality, including savings in time, savings 
in the cost of materials, and the possibility of examining multiple models before deciding which 
model is most suitable for an intended purpose. However, in providing a representation that 

15 involves only two of the three dimensions of an object, information is lost, such as visual and 
tactile clues that would normally be available to the modeler who works on a real object. The 
methods and systems of the invention provide solutions that restore the lost information to the 
modeler or user of the system. A description of exemplary modeling environment equipment 
with which the methods and systems of the invention may be practiced will first be presented, 

20 and the detailed description of the methods and systems of the invention will follow. 

[0049] Fig. 1 is a schematic perspective view of an exemplary six degree of freedom 

force reflecting haptic interface 10 that can be used in accordance with one embodiment of the 
invention. The interface 10 can be used by a user to provide input to a device, such as a 
computer, and can be used to provide force feedback from the computer to the user. The six 

25 degrees of freedom of interface 10 are independent. 

[0050] The interface 10 includes a housing 12 defining a reference ground, six joints or 

articulations, and six structural elements. A first powered tracked rotary element 14 is supported 
by the housing 12 to define a first articulation 16 with an axis "A" having a substantially vertical 
orientation. A second powered tracked rotary element 18 is mounted thereon to define a second 

30 articulation 20 with an axis "B" having a substantially perpendicular orientation relative to the 



10 



space. 



[0048] 



There are many advantages to be gained by performing operations in a modeling 
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first axis, A, A third powered tracked rotary element 22 is mounted on a generally outwardly 
radially disposed extension 24 of the second element 18 to define a third articulation 26 having 
an axis "C" which is substantially parallel to the second axis, B. A fourth free rotary element 28 
is mounted on a generally outwardly radially disposed extension 30 of the third element 22 to 
5 define a fourth articulation 32 having an axis "D" which is substantially perpendicular to the 
third axis, C. A fifth free rotary element 34 is mounted on a generally outwardly radially 
disposed extension 36 of the fourth element 28 to define a fifth articulation 38 having an axis "E" 
which is substantially perpendicular to the fourth axis, D. Lastly, a sixth free rotary user 
connection element 40 in the form of a stylus configured to be grasped by a user is mounted on a 

1 0 generally outwardly radially disposed extension 42 of the fifth element 34 to define a sixth 

articulation 44 having an axis "F" which is substantially perpendicular to the fifth axis, E. The 
haptic interface of Fig. 1 is fully described in commonly-owned U.S. patent application serial No. 
09/356,1 19, filed July 16, 1999, which application is incorporated by reference herein in its 
entirety. Those familiar with the haptic arts will recognize that there are many different haptic 

1 5 interfaces that convert the motion of an object under the control of a user to electrical signals, 
many different haptic interfaces that convert force signals generated in a computer to mechanical 
forces that can be experienced by a user, and haptic interfaces that accomplish both results. 
[0051] Fig. 2 illustrates an exemplary general purpose computer 200 that can be used in 

accordance with one embodiment of the present invention, such as a commercially available 

20 personal computer that includes a CPU, one or more memories, one or more storage media, one 
or more output devices, such as a display 210, and one or more input devices, such as a keyboard 
220. The computer operates using any commercially available operating system, such as any 
version of the Windows™ operating systems from Microsoft Corporation of Redmond, 
Washington, or the Linux™ operating system from Red Hat Software of Research Triangle Park, 

25 North Carolina. In some embodiments, a haptic device such as the interface 10 is present and is 
connected for communication with the computer 200, for example with wires. In other 
embodiments, the interconnection can be a wireless or an infrared interconnection. The interface 
10 is available for use as an input device and/or an output device. The computer is programmed 
with software including commands that, when operating, direct the computer in the performance 

30 of the methods of the invention. Those of skill in the programming arts will recognize that some 
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or all of the commands can be provided in the form of software, in the form of programmable 
hardware such as flash memory, ROM, or programmable gate arrays (PGAs), in the form of hard- 
wired circuitry, or in some combination of two or more of software, programmed hardware, or 
hard- wired circuitry. Commands that control the operation of a computer are often grouped into 
5 units that perform a particular action, such as receiving information, processing information or 
data, and providing information to a user. Such a unit can comprise any number of instructions, 
from a single command, such as a single machine language instruction, to a plurality of 
commands, such as a plurality of lines of code written in a higher level programming language 
such as C++. Such units of commands will be referred to generally as modules, whether the 

10 commands include software, programmed hardware, hard-wired circuitry, or a combination 
thereof. The computer and/or the software includes modules that accept input from input 
devices, that provide output signals to output devices, and that maintain the orderly operation of 
the computer. In particular, the computer includes at least one data input module that accepts 
information from the interface 10 which is indicative of the state of the interface 10 and its 

1 5 motions. The computer also includes at least one module that renders images and text on the 
display 210. 

[0052] In alternative embodiments, the computer 200 is a laptop computer, a 

minicomputer, a mainframe computer, an embedded computer, or a handheld computer. The 
memory is any conventional memory such as, but not limited to, semiconductor memory, optical 

20 memory, or magnetic memory. The storage medium is any conventional machine-readable 
storage medium such as, but not limited to, floppy disk, hard disk, CD-ROM, and/or magnetic 
tape. The display is any conventional display such as, but not limited to, a video monitor, a 
printer, a speaker, an alphanumeric display, and/or a force-feedback haptic interface device. The 
input device is any conventional input device such as, but not limited to, a keyboard, a mouse, a 

25 force-feedback haptic interface device, a touch screen, a microphone, and/or a remote control. 
The computer can be a stand-alone computer or interconnected with at least one other computer 
by way of a network. 

[0053] Fig. 3 is an illustrative flow diagram 300 that shows the steps of one embodiment 

of the method of the invention. The figure also shows the flow of control among the modules of 
30 one embodiment of the system of the invention. The method begins at oval 302. The three- 
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dimensional modeling application is initialized as a result of the operation of the modeling 
module, as indicated by box 304. The modeling application is populated with at least one virtual 
object and a three-dimensional cursor, as a result of the operation of modeling module, as 
indicated at box 306. The first position of the three-dimensional cursor is determined, as a result 
5 of the operation of the cursor tracking module, as indicated by box 308. As is described below in 
greater detail, the cursor position is determined in relation to the position of the interface 10 in 
real space, using such apparatus as position sensors, analog to digital converters, and buffers or 
similar locations in machine-readable memory to store values indicative of a location in space 
and a spatial orientation. In some embodiments, the initial location and orientation can be 

1 0 defined by default values. 

[0054] One or more virtual objects that have been input or generated in the three- 

dimensional modeling environment are displayed in one view of a two-dimensional display, such 
as display 210, as a result of the operation of a rendering module, as indicated in box 310. 
Buffers or memory locations in the computer memory are used to store information indicative of 

15 the position and orientation of each virtual object in machine-readable memory. The position of 
the three-dimensional cursor is represented in the same view on the two-dimensional display, as a 
result of the operation of the rendering module, as indicated in box 312. 
[0055] The user selects a virtual object, as indicated at box 3 14, by manipulating the 

three-dimensional cursor which is responsive to the motions of the interface 10 so as to coincide 

20 with the virtual object in two of three dimensions, such as the two dimensions of the two- 
dimensional display, and by issuing a selection command. The selection command can be any 
action that the computer is capable of sensing and that it is programmed to recognize, such as, for 
example, pushing a button, making an audible sound, or pressing a keyboard key. The selection 
process is monitored by the selection module. 

25 [0056] The user can carry out different actions with regard to the selected virtual object. 

The user can edit the virtual object, as a result of user interaction with the editing module, as 
indicated at box 316. Editing an object can include, but is not limited to, coloring an object, 
modifying an object, combining an object with one or more other objects, grouping the object 
with one or more other objects, deleting the object, renaming the object, saving the object to a 



EXPRESS MAH^ABELNo.: EL934410273US 
Attorney Docket No.: SNS-010 (7268/-) 

-14- 

file, and performing other file functions that involve the object. In a preferred modeling 
application embodiment, editing the object involves sculpting the virtual object. 
[0057] The user can also manipulate an object, whether the object is edited or not. The 

order of performing an edit and a manipulation is not restricted. In the flow diagram of Fig. 3, a 
5 bi-directional arrow connecting boxes 316 and 320 indicates that either action can precede or can 
follow the other, at the user's discretion. 

[0058] The user can select a local origin point on the selected virtual object. The local 

origin point can be selected in an arbitrary manner from any point that is included in the selected 
virtual object. The user performs the selection of the local origin point in conjunction with the 
10 object tracking module, which monitors the selection process. The selection process is indicated 
at box 320. 

[0059] When the local origin point and the position of the three-dimensional cursor are 

defined, the system of the invention defines a mathematical transformation. The transformation 
is a mathematical relationship that represents a difference in location of the local origin point and 
15 the position of the three-dimensional cursor at the time when the user selects the local origin 
point on the virtual object. The step of defining the mathematical transformation, which is 
performed by the transformation module, is indicated at box 322. 

[0060] The user can manipulate the selected virtual object, which can be performed in 

conjunction with an object manipulation module that manipulates the selected object in 

20 correspondence with manipulations of the interface 10 by the user. It is more convenient for the 
user to see the manipulation with the three-dimensional cursor displayed at the position where 
the manipulation is applied to the virtual object. According to principles of the invention, the 
three-dimensional cursor is repositioned to coincide with the local origin point by application of 
the transformation to the position of the three-dimensional cursor, as indicated at box 324 of the 

25 flow diagram. The application of the transformation is performed by the transformation module, 
and a relocation module performs the relocation of the cursor, as indicated in the flow diagram at 
box 332. 

[0061] The methods and systems of the invention also provide optional visual aids and 

optional haptic aids that can be of use to the user in determining relative locations and relative 
30 motions of the three-dimensional cursor with respect to objects. The optional visual and haptic 
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aids are also useful with respect to recognizing relative position and relative motion of the three- 
dimensional cursor with respect to features of the three-dimensional modeling environment, such 
as locations, directions, planes, and volumes. The provision and operation of a visual aid module 
that defines and controls visual aids and of a haptic aid module that defines and controls haptic 
5 aids are denoted by boxes 328 and 330, respectively. 

[0062] The selection and manipulation process is completed when the user releases an 

object, having selected an object and having performed editing and/or manipulation operations. 
The user can release the object by issuing a command recognized by the system as being a 
command to release the then-selected object, such as a key press or the like. The relocation 

10 module can reposition the cursor at the location that it would have if the manipulations that the 
user performed on the selected object had been performed directly on the position of the three- 
dimensional cursor. In other terms, the three-dimensional cursor can be relocated to the same 
position relative to the position of the interface 10 at the time the release command is issued as it 
had relative to the position of the interface 10 at the time the selection command was issued. The 

15 selection and manipulation process can be repeated as many times as the user wishes, as 

indicated by the loop back from box 332 to box 314. On any pass around the system, the user 
can select to perform a null selection at any step (i.e., do nothing, leave everything as it is) and 
proceed on to the next user-influenced step in the loop. When the user wishes, the user can cause 
the system and method to terminate operation at the oval End 334. 

20 [0063] Figs. 4 through 8 each show two panels. In each figure, the panel on the left is an 

exemplary view that a user sees on a conventional two-dimensional display, such as display 210 
of Fig. 2. The panel on the right of each figure is an optional additional exemplary view in a 
plane different from the plane of the panel on the left. In some embodiments, the panel on the 
right corresponds to a plane that is orthogonal to the plane of the view on the left panel. In some 

25 embodiments, the additional view may optionally be provided to a user. The additional view is 
useful to explain the methods and systems of the invention. In the figures presented here, the 
additional view is shown at a smaller scale than the view in the left panel. 
[0064] Fig. 4 shows an illustrative three dimensional virtual object 425 that is generated 

in a three-dimensional modeling environment, along with a three dimensional cursor 435. The 

30 three-dimensional modeling environment is generated by a modeling module that, when it is 
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operating on the computer 200, initializes a three-dimensional modeling space, and that has the 
capacity to contain one or more virtual objects and a three-dimensional cursor. The three 
dimensional virtual object 425 is displayed in two dimensions on a two-dimensional display 210. 
The three dimensional cursor 435 indicates a position in the three-dimensional modeling 
5 environment corresponding to a position of the haptic interface 10 that is controlled in real space 
by the user. In various embodiments, the three-dimensional virtual object is a selected one of a 
point, a straight line segment, a curvilinear segment, a spline, a two-dimensional representation, 
and a three dimensional representation. As indicated in the left panel 405 of Fig. 4, the 
coordinates of the three-dimensional cursor 435 in the two-dimensional display 210 are close to 

10 coordinates corresponding to a point 445 of object 425. However, as is seen in the panel on the 
right 415, the third dimensional coordinate of the three-dimensional cursor 435 differs 
considerably from the third coordinate of the point 445 along the third dimension of the three 
dimensional display space. This difference highlights one of the problems that the methods and 
systems of the invention overcome. 

1 5 [0065] Fig. 5 shows an illustrative example of a circumstance wherein the user brings the 

position of the interface 10 into collocation with the intended point 445 of object 425. Fig. 5 
shows an illustrative diagram in which the three-dimensional cursor 435 is collocated with a 
point 445 of the two dimensional object 425 in all three dimensions, as shown in the two 
dimensional display 210 on the left panel 505, and in the panel on the right 515. However, if the 

20 user has only the visual clues provided by the left panels of Figs. 4 and 5 as guidance, the user is 
hard-pressed to determine either that the position of the three-dimensional cursor 435 in Fig. 5 is 
at the desired location, or that the position of the three dimensional cursor 435 in Fig. 4 is far 
from the correct position along the third dimension. 

[0066] Fig. 6 shows an illustrative example in which the user has controlled the position 

25 of the interface 10 so as to bring the three-dimensional cursor 435 near coincidence in two 
dimensions with a point 445 on object 425, as shown in panel 605 on the left. The optional 
additional view in the right hand panel 615 shows an offset distance d along the dimension 
perpendicular to the plane of the view of panel 605, between the three dimensional cursor 435 
and the point 445. The user, by controlling the interface 10, causes the three dimensional cursor 
30 435 to move upward in the direction of the arrow 606 in panel 605. The three-dimensional 
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cursor 435 in panel 615 moves in the same direction in response to the user controlling the 
interface 10. If a plurality of virtual objects are displayed on display 210, the user can select any 
point of any virtual object that is visible to the user. 

[0067] Fig. 7 shows an illustrative example of the operation of the methods and systems 

5 of the invention when the three-dimensional cursor 435 reaches the location of the point 445 of 
object 425 in two dimensions as displayed on two dimensional display 210. At such time that the 
three-dimensional cursor 435 is collocated with the point 445, the user can issue a command, for 
example by pressing a button on a mouse or on the haptic interface, depressing a key on a 
keyboard, making an audible sound, or some equivalent action. The system recognizes the 

10 command, and causes the selection of object 425. The user can pick the point 445 arbitrarily 
from any point on the virtual object 425. In Fig. 7, selected virtual object 425 is shown in a 
somewhat heavier weight line than is the unselected virtual object 425 in Fig. 6. The viewer/user 
can observe the selection by a visual change in the virtual object 425 as it is displayed on the 
two-dimensional display 210. The point 445 is treated as a local origin on the selected virtual 

1 5 object 425. The local origin point 445 can be represented for example in a computer memory as 
a discrete array that contains three positional coordinates and three angular coordinates, which 
values are initialized at the time of selection of virtual object 425. 

[0068] At the same time, the location of three-dimensional cursor 435 is brought into 

three-dimensional collocation with the point 445, by a transformation module operating on the 

20 computer 200 that determines a transformation in the three-dimensional modeling environment. 
In the right panel 715 of Fig. 7, the transformation is a vector extending from the position 435' of 
the three-dimensional cursor at the time the user issues the selection command to the location 
435 on object 425. For the example shown in Fig. 7, the transformation is substantially a vector 
having a component directed orthogonal to the two-dimensional display space of panel 705. In 

25 some embodiments, the transformation can include vectorial components corresponding to each 
of the dimensions of the modeling space, and can also include angular displacements, or 
rotations, centered at either or both of the position in the three dimensional modeling space of the 
input device, such as interface 10, or the position of the local origin point 445. 
[0069] As will be apparent to those of skill in the arts relating to mathematical 

30 transforms, there can be representations of three-dimensional space in a variety of different 
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mathematical representations that can be shown to be equivalents of each other. For convenience 
of exposition, the present description is given using conventional Cartesian coordinates having 
three mutually perpendicular rectilinear dimensions. Equivalently, one could represent the three- 
dimensional modeling space using other coordinate systems such as polar coordinates, cylindrical 
5 coordinates, or another coordinate system, as desired. 

[0070] There are many purposes for selecting a virtual object in the three dimensional 

modeling space. For example, the user can select an object for editing. An editing module 
within the computer performs editing functions in response to commands from the user. 
[0071] Fig. 8 is a diagram that shows an illustrative manipulation of a virtual object 

10 combined with application of a transform. As shown in the right panel 815 of Fig. 8, the three- 
dimensional cursor 435 is collocated with the selected local origin point 445 of virtual object 425 
as a result of the application of the transformation determined by the transformation module. 
The user can manipulate the interface 10 as an input device, in order to manipulate the virtual 
object 425. As shown in the left panel 805, when, for example, the user moves the input device 

15 10 in a substantially downward direction, the motion of the input device 10 is tracked by a cursor 
tracking module within the computer. The cursor tracking module determines a location of the 
three-dimensional cursor 435 within the three-dimensional modeling environment, which 
position corresponds to a position of the input device 1 0 as it is manipulated by the user. For a 
substantially downward and slightly sideward motion of the input device 10, as indicated by the 

20 arrow 820, the position of the cursor 435 follows the motion of the input device 10 to the 

location given by 435". In addition, the object 425 is transformed to a shape corresponding to 
that of object 425' . In the embodiment shown, the locus of object 425 has been constrained to 
pass through points A, B and C, so object 425" deviates from object 425 between point 435" and 
point C and between points B and C, but not at points B and C themselves. In one embodiment, 

25 the cursor tracking module is initiated when the modeling environment starts to operate, and 
enunciators within the input device 10 send signals to the cursor tracking module that allow the 
module to follow the users' manipulations of the input device 10 in both translational and 
rotational degrees of freedom. 

[0072] An object tracking module follows the motions of each virtual object that is 

30 created within the three-dimensional modeling environment. In one embodiment, an object 



EXPRESS MAIMftBEL No.: EL934410273US 
Attorney Docket No.: SNS-010 (7268/-) 

-19- 

tracking module is initialized when a virtual object 425 is created within the three-dimensional 
modeling environment. The object tracking module receives input corresponding to the 
manipulation of the input device 10 by the user, and updates that location of the virtual object 
425. In one embodiment, the object tracking module tracks the location of the local origin point 
5 445 on the selected virtual object 425. 

[0073] A relocation module applies the transformation to the three-dimensional cursor 

435. In one embodiment, the relocation module applies that transformation only during the time 
period when the virtual object 425 is selected. During the period when the mathematical 
transformation has been applied to the three-dimensional cursor 435, the three-dimensional 

10 cursor 435 appears at the location of the local origin point 445. The user is provided a visual aid 
in manipulating the virtual object 425, in that a deviation in location between the virtual object 
425 and the three-dimensional cursor 435, and equivalently, the third dimensional location of the 
input device 10, is eliminated temporarily while the user manipulates the virtual object 425. The 
user can manipulate the virtual object 425 in correspondence with the manipulation of the input 

15 device 10, such as the haptic interface. In the frame of reference that the user experiences, it is as 
if the user was grasping the virtual object directly, rather than manipulating it over a distance. 
The user can manipulate any of the degrees of freedom of the input device 10, individually, in 
any combination, or all simultaneously. In an embodiment employing the six degree of freedom 
input device depicted in Fig. 1, the user can manipulate one, two, three, four, five, or all six 

20 degrees of freedom in any combination and at any time. 

[0074] Other visual aids can be provided for the user. In some systems, constraints that 

limit a point or a motion in some way can be implemented, by the expedient of invoking one or 
more suitable constraint equations. Examples of constraints that can be applied to a visual 
representation of a three-dimensional model include limiting a point to move along a specific 

25 locus or plane. In one embodiment, the point can be constrained to move along axial directions 
that are aligned to an axis of the three-dimensional modeling environment. In some 
embodiments, the constraint to move along an axial direction may be applied in response to a 
command from a user. In another embodiment, visual aids that indicate the location of a point 
along an axial direction, along a line, on a plane, or within a volume are provided to assist a user 

30 in visualizing relationships between objects, modifications of objects, and motions of objects. In 
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some embodiments, the visual aid can be a context-specific visual aid consistent with a user- 
defined geometrical limitation, such as the constraint by a user that a surface remain smooth 
during manipulation, or that a surface remain convex. 

[0075] When the user desires to cease manipulating the selected virtual object 425, the 

5 user can issue a command, such as repeating the command issued to select the virtual object 425. 
Alternatively, the user can issue another command specifically available to indicate to the 
computer 200 that manipulation of the selected object 425 is complete, at least for the moment. 
Upon issuance of such a command, the cursor tracking module moves the three-dimensional 
cursor 435 to the location that it would have if the manipulation of the input device 1 0 by the 

10 user had been applied directly to the three-dimensional cursor. There are numerous ways in 
which such a result can be attained, as will be apparent to those skilled in the art. One is, as 
indicated, the direct manipulation of the location of the three-dimensional cursor 435 beginning 
with its position at the start of manipulation in parallel with the manipulation of the virtual object 
425, followed by a rendering of the three-dimensional cursor 435 at the final location when the 

1 5 command to cease manipulation of the selected virtual object is issued. Another is the 

application of the inverse of the transformation originally applied to the three-dimensional cursor 
to the final position of the three-dimensional cursor. Yet another is the application of the inverse 
of the transformation originally applied to the three-dimensional cursor to the final position of 
the local origin point 445, which moves with the three-dimensional cursor during the 

20 manipulation. Still other computations can be performed that produce an equivalent result. 
[0076] The haptic input device 1 0 may be available for use throughout the process of 

selecting, editing, manipulating, and releasing a virtual object, as well as during operation of 
visual display operations of the computer system, including the provision of visual aids to the 
user. The haptic input device 10 is also capable of providing haptic force feedback to the user in 

25 any one or more of all six degrees of freedom. In some embodiments, the haptic force feedback 
function provides yet another aid, namely a haptic aid, to the user. In some embodiments, the 
haptic force feedback device 10 provides dynamic frictional forces during a positional 
correspondence of the virtual object and the cursor in the two-dimensional display space, for 
example, to indicate the correspondence to the user. 
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[0077] The haptic aids can also be applied in concert with visual aids. In one 

embodiment, when the user-activated visual constraint limiting a point to a locus aligned to an 
axis of the three-dimensional modeling environment is operating, the haptic aid includes a haptic 
constraint, limiting motion of the three-dimensional cursor to directions aligned to an axis of the 
5 three dimensional environment, except within a region of radius R about an identified point. The 
dimension R depends on the precision that the user wants to obtain. An exemplary radius is 25 
pixels. In one embodiment, the haptic aid is contemporaneously accompanied by a display of a 
visual aid component that indicates an axial location of the cursor along an axis. 
[0078] In one embodiment, when the user-activated visual constraint limiting a point to a 

10 selected line is operating, the haptic aid includes a haptic constraint limiting motion of the three- 
dimensional cursor to the line. In one embodiment, the haptic aid is contemporaneously 
accompanied by a display of a visual aid component that indicates the location of an axial 
location of the cursor along an axis. 

[0079] In one embodiment, when the user-activated visual constraint limiting a point to a 

15 selected plane is operating, the haptic aid includes a haptic constraint limiting motion of the 

three-dimensional cursor to the plane. In one embodiment, the haptic aid is contemporaneously 
accompanied by a display of a visual aid component that indicates the location of the plane. 
[0080] Fig. 9 is a drawing that illustrates an embodiment in which a selection of a virtual 

object 425 is depicted when a display 21 0 is operating in an orthogonal view mode, without 
20 perspective. In Fig. 9, a wire frame 910, 920 appears as a simple two-dimensional line drawing. 
Fig. 10 is a drawing that illustrates an embodiment in which a selection of a virtual object 425 is 
depicted when a display 210 is operating in a perspective view mode. Fig. 10 shows 
substantially the same view as Fig. 9, with the inclusion of visual aids in the form of a wire frame 
1010, 1020 shown in perspective. As described above, in the two upper images of each figure, 
25 the selection has not yet been made, and the three-dimensional cursor 435 is not coincident with 
a virtual object 425. In the lower images, the three-dimensional cursor 435 is coincident with a 
virtual object 425, which appears in heavier weight. As described above, the left panels depict a 
first view of the virtual objects and the three-dimensional cursor, and the right panels depict an 
optional second view having a view plane different from the view plane of the panels on the left. 
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[0081] In the situation where a first virtual object occludes a second virtual object from 

the view of the user, the user can select the first virtual object and move it, or can select it and 
erase it, so as to leave the second virtual object visible. In similar manner, any virtual object can 
be iteratively selected, manipulated, released, selected again at the same or at a different point on 
the object, manipulated again, released again, and so forth as many times as the user desires. 



Equivalents 

[0082] While the invention has been particularly shown and described with reference to 

specific preferred embodiments, it should be understood by those skilled in the art that various 
10 changes in form and detail may be made therein without departing from the spirit and scope of 
the invention as defined by the appended claims. 
[0083] What is claimed is: 



